{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "77ad8f22-9026-47d1-b7a7-43360858b197",
   "metadata": {},
   "source": [
    "# Evaluating MMLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7ba248ad-5990-4430-b992-db065b053b13",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "0ae0bf41-23bb-4fca-9f43-1f5d71a8498c",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_dir = \"/home/jiw203/wjn/InstructGraph/output/instruction_tuning/fsdp_peft_flash_1500k/llama2-peft-2epoch/mmlu_predictions/all_prediction.json\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "65e4c3ab-ebd2-45e6-8c2c-fc0ef04e6e1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_dir = \"/home/jiw203/wjn/InstructGraph/output/instruction_tuning/fsdp_peft_flash_1500k/vicuna-peft-2epoch/mmlu_predictions/all_prediction.json\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "fa051601-edb8-4f4b-8829-1603bf526aaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_dir = \"/home/jiw203/wjn/InstructGraph/output/instruction_tuning/llama2/mmlu_predictions/all_prediction.json\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "1452b1da-13f3-4aea-a513-7c124efdf16a",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction_dir = \"/home/jiw203/wjn/InstructGraph/output/instruction_tuning/vicuna/mmlu_predictions/all_prediction.json\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "f1c8dff0-1d03-4b9e-b89f-f8776e256409",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(prediction_dir, \"r\", encoding=\"utf-8\") as fr:\n",
    "    predictions = json.load(fr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "ae97bfc2-fcac-4d09-a209-000cf1dbcdb6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "900"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "3ccb9aa6-2bf9-4629-8d3d-c91a9b83e0c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(predictions):\n",
    "    acc = 0\n",
    "    for ei, example in enumerate(predictions):\n",
    "        prediction = \"\\n\\n\" + example[\"prediction\"]\n",
    "        answer = example[\"answer\"][0]\n",
    "        prediction = \"\".join(prediction.split(\"\\n\\nQ\")[6].split(\"\\n\")[-1])\n",
    "        # print(prediction)\n",
    "        if \"{}\".format(answer) in prediction:\n",
    "            acc += 1.0\n",
    "        # else:\n",
    "        #     print(\"{} - {} - {}\".format(ei, answer, prediction))\n",
    "            # assert 1>2\n",
    "    print(\"acc={}\".format(round(acc / len(predictions), 4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "94a44561-6c23-46c0-b130-c828653fc847",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc=0.5567\n"
     ]
    }
   ],
   "source": [
    "evaluate(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "13ead913-d39e-493c-8e36-5fc1773e0c72",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'task_name': 'high_school_computer_science',\n",
       " 'idx': 96,\n",
       " 'prediction': 'Q: Consider the following list. - Assembly language - Block-based programming language - Logic gate - Machine language Which of the following arranges the list in order from highest level of abstraction to lowest level of abstraction?\\nOptions:\\n(A): Block-based programming language, assembly language, machine language, logic gate\\n(B): Block-based programming language, machine language, assembly language, logic gate\\n(C): Block-based programming language, machine language, logic gate, assembly language\\n(D): Machine language, block-based programming language, assembly language, logic gate\\nA: The answer is (A).\\n\\nQ: Which types of functions grow the slowest?\\nOptions:\\n(A): O(N^(1/2))\\n(B): O(N^(1/4))\\n(C): O(N^(1/N))\\n(D): O(N)\\nA: The answer is (C).\\n\\nQ: A large hospital maintains a list of patients’ records in no particular order. To find the record of a given patient, which represents the most efficient method that will work?\\nOptions:\\n(A): Do a sequential search on the name field of the records.\\n(B): Do a binary search on the name field of the records.\\n(C): Use insertion sort to sort the records alphabetically by name; then do a sequential search on the name field of the records.\\n(D): Usemergesort to sort the records alphabetically by name; then do a sequential search on the name field of the records.\\nA: The answer is (A).\\n\\nQ: In Python 3, let r = lambda q: q * 2. What is r(3)?\\nOptions:\\n(A): 2\\n(B): 6\\n(C): 3\\n(D): 1\\nA: The answer is (B).\\n\\nQ: Which is a valid expression in Python 3.5?\\nOptions:\\n(A): sort(\\'ab\\')\\n(B): sorted(\\'ab\\')\\n(C): \"ab\".sort()\\n(D): 1/0\\nA: The answer is (B).\\n\\nQ: Consider the following code segment, which uses the variables r, s, and t.\\n  r ← 1\\n   s ← 2\\n   t ← 3\\n   r ← s\\n   s ← t\\n   DISPLAY (r)\\n   DISPLAY (s)\\n\\n What is displayed as a result of running the code segment?\\nOptions:\\n(A): 1 1\\n(B): 1 2\\n(C): 2 3\\n(D): 3 2\\nA: The answer is (B).',\n",
       " 'answer': ['(C)']}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions[10440]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "64dd9398-8fd3-459c-b3a3-4d8fb3d968c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./mmlu_dataset.json\", \"r\", encoding=\"utf-8\") as fr:\n",
    "    datasets = [json.loads(i) for i in fr.readlines()]\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "7d74c865-7dd2-471d-92cf-d55c861314c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Q: This question refers to the following information.\n",
      "Questions to the passage below, an account given to a French officer in Algeria in the 1830s by a member of an Arab slave trade caravan.\n",
      "The Slave Trade\n",
      "All of you [soldiers], are summoned . . . to hunt the idolatrous Koholanes [a pejorative word for \"black Africans\"]. . . . The soldiery divided themselves into two companies . . . with orders to attack places without defenses and to carry off the inhabitants as well as seizing all peasants busy cultivating their fields. . . . Whilst waiting for the return of the companies despatched to hunt Negroes, we went every day to the slave market where we bought at the following prices:\n",
      "A Negro with beard………………10 or 15,000 cowries.\n",
      "They are not considered as merchandise since one has little chance of preventing them from escaping.\n",
      "An adult Negress………………..10 or 15,000 cowries for the same reasons\n",
      "An adolescent Negro………………….30,000 cowries\n",
      "A young Negress ……………………50–60,000 cowries\n",
      "The price varies according to whether she is more or less beautiful.\n",
      "A male Negro child……………………..45,000 cowries\n",
      "A female Negro child ……………………….35–40,000 cowries\n",
      "Finally, our caravan which had set out from Algeria with sixty-four camels and only sixteen persons, was now augmented by four hundred slaves, of whom three hundred were women. . . . It was at this point that suddenly a confused noise of cries and sobs passed from one group of slaves to another and reached our own. . . . Some rolled on the ground, clung to bushes and absolutely refused to walk. . . . They could only be got up with mighty lashes of the whip and by rendering them completely bloody.\n",
      "What is the most likely destination for the captured slaves in the excerpt?\n",
      "Options:\n",
      "(A): Elites' homes or harems in the Middle East\n",
      "(B): Sugar plantations in Brazil\n",
      "(C): Cotton plantations in North America\n",
      "(D): Slave armies of the Mughal Empire\n",
      "A: The answer is (A).\n",
      "\n",
      "Q: This question refers to the following information.\n",
      "Source 1:\n",
      "\"The case of a broken thigh is analogous to that of the arm, but in particular, a fractured thigh is mostly deranged forwards and outwards, for the bone is naturally flattened on those sides. It is to be set by the hands, with ligatures, and even cords applied, the one above and the other below the fracture. When the fracture takes place at one end, if at the head of the thigh, the middle part of a thong wrapped round with wool, so that it may not cut the parts there, is to be applied to the perineum, and the ends of it brought up to the head and given to an assistant to hold, and applying a ligature below the fracture, we give the ends of it to another assistant to make extension. If it is fractured near the knee, we apply the ligature immediately above the fracture, and give the ends to an assistant, with which to make extension upwards; and while we put a ligature round the knee to secure it, and while the patient lies thus, with his leg extended, we arrange the fracture.\"\n",
      "Paul of Aegina, Epitome: On the Fracture of the Thigh and Nose, late seventh century C.E.\n",
      "Source 2:\n",
      "\"Medicine considers the human body as to the means by which it is cured and by which it is driven away from health. The knowledge of anything, since all things have causes, is not acquired or complete unless it is known by its causes. Therefore in medicine we ought to know the causes of sickness and health. And because health and sickness and their causes are sometimes manifest, and sometimes hidden and not to be comprehended except by the study of symptoms, we must also study the symptoms of health and disease. Now it is established in the sciences that no knowledge is acquired save through the study of its causes and beginnings, if it has had causes and beginnings; nor completed except by knowledge of its accidents and accompanying essentials.\"\n",
      "Ibn Sina (Avicenna), On Medicine, ca. 1020 C.E.\n",
      "Which of the following characterizes both of the passages?\n",
      "Options:\n",
      "(A): A reliance upon supernatural understandings of physical phenomena\n",
      "(B): A reliance upon reason and rationalistic understandings of physical phenomena\n",
      "(C): A reliance upon abstract philosophical understandings of physical phenomena\n",
      "(D): A reliance upon astrological understandings of physical phenomena\n",
      "A: The answer is (B).\n",
      "\n",
      "Q: This question refers to the following information.\n",
      "An Act to place certain restrictions on Immigration and to provide for the removal from the Commonwealth of Prohibited Immigrants.\n",
      "…\n",
      "3. The immigration into the Commonwealth of the persons described in any of the following paragraphs in this section (hereinafter called \"prohibited immigrants\") is prohibited, namely\n",
      "(a) Any person who when asked to do so by an officer fails to write out at dictation and sign in the presence of the officer a passage of fifty words in length in a European language directed by the officer;\n",
      "(b) Any person in the opinion of the Minister or of an officer to become a charge upon the public or upon any public or charitable organisation;\n",
      "…\n",
      "(g) Any persons under a contract or agreement to perform manual labour within the Commonwealth: Provided that this paragraph shall not apply to workmen exempted by the Minister for special skill required by Australia…\n",
      "Immigration Restriction Act of 1901 (Australia)\n",
      "Whereas in the opinion of the Government of the United States the coming of Chinese laborers to this country endangers the good order of certain localities within the territory thereof;\n",
      "Therefore,\n",
      "Be it enacted by the Senate and House of Representatives of the United States of America in Congress assembled, That from and after the expiration of ninety days next after the passage of this act, the coming of Chinese laborers to the United States be suspended; and during such suspension it shall not be lawful for any Chinese laborer to come, or having so come after the expiration of said ninety days to remain within the United States.\n",
      "SEC. 2. That the master of every vessel who shall knowingly bring within the United States on such vessel, and land or permit to be landed, any Chinese laborer, from any foreign port or place, shall be deemed guilty of a misdemeanor, and on conviction thereof shall be punished by a fine of not more than five hundred dollars for each and every such Chinese laborer so brought, and may be also imprisoned for a term not exceeding one year….\n",
      "Chinese Exclusion Act of 1882 (United States)\n",
      "What broad trend from the 1800s formed the background for the enactment of the laws described in both passages?\n",
      "Options:\n",
      "(A): The growing appeal of Christianity among Indian and Chinese populations\n",
      "(B): European and American colonial domination of India and China\n",
      "(C): The Pacific-wide migration of Indian and Chinese indentured laborers\n",
      "(D): Gradual industrialization of the Indian and Chinese economies\n",
      "A: The answer is (C).\n",
      "\n",
      "Q: This question refers to the following information.\n",
      "Meng Yizi asked about the treatment of parents. The Master said, \"Never disobey!\" When Fan Chi was driving his carriage for him, [he asked], \"In what sense did you mean this?\" The Master said, \"While they are alive, serve them according to proper ritual. When they die, bury them according to proper ritual, and sacrifice to them according to proper ritual.\"\n",
      "Confucius, The Analects, Book 2, Chapter 5\n",
      "Which of the following ideals does the passage above exemplify?\n",
      "Options:\n",
      "(A): Filial piety\n",
      "(B): Forgiveness of wrongdoings\n",
      "(C): Respect for honesty and moderation\n",
      "(D): Monotheistic worship\n",
      "A: The answer is (A).\n",
      "\n",
      "Q: This question refers to the following information.\n",
      "\"The quicksilver mines of Huancavelica are where the poor Indians are so harshly punished, where they are tortured and so many Indians die; it is there that the noble caciques [headmen] of this kingdom are finished off and tortured. The same is true in all the other mines: the silver mines of Potosi [and others]….The owners and stewards of the mines, whether Spaniards, mestizos, or Indians, are such tyrants, with no fear of God or Justice, because they are not audited and are not inspected twice a year….\n",
      "And they are not paid for the labor of traveling to and from the mines or for the time they spend at the mines. The Indians, under the pretext of mining chores, are made to spend their workdays herding cattle and conveying goods; they are sent off to the plains, and the Indians die. These Indians are not paid for their labor, and their work is kept hidden.\n",
      "And [the mine owners] keep Indian cooking women in their residences; they use cooking as a pretext for taking concubines….And they oblige the Indians to accept corn or meat or chicha [corn beer]…at their own expense, and they deduct the price from their labor and their workdays. In this way, the Indians end up very poor and deep in debt, and they have no way to pay their tribute.\n",
      "There is no remedy for all this, because any [colonial official] who enters comes to an agreement with the mine owners, and all the owners join forces in bribing him….Even the protector of the Indians is useless;…he [does not] warn Your Majesty or your royal Audiencia [court] about the harms done to the poor Indians.\"\n",
      "Excerpt from The First New Chronicle and Good Government [abridged], by Felipe Guaman Poma de Alaya. Selected, translated, and annotated by David Frye. Copyright 2006 Hackett Publishing Company. Reprinted with permission from the publisher.\n",
      "Felipe Guaman Poma de Ayala, The First New Chronicle and Good Government, ca. 1610\n",
      "Which of the following conclusions about the period 1450–1750 C.E. is most directly supported by the passage?\n",
      "Options:\n",
      "(A): The population of the Americas declined.\n",
      "(B): Silver flows from the Americas went principally to Europe and Asia.\n",
      "(C): The religious traditions and social customs of the New World were dramatically altered by the arrival of Europeans.\n",
      "(D): Intermarriage between Spanish colonists and natives led to equality between the races.\n",
      "A: The answer is (A).\n",
      "\n",
      "Q: This question refers to the following information.\n",
      "This great purity of the French Revolution is precisely what causes both our strength and our weakness. Our strength, because it gives to us rights of the public interest over private interests; our weakness, because it rallies all vicious men against us. We must smother the internal and external enemies of the Republic or perish with it; now in this situation, the first maxim of your policy ought to be to lead the people by reason and the people's enemies by terror. Terror is nothing other than justice, prompt, severe, inflexible; it is therefore an emanation of virtue; it is not so much a special principle as it is a consequence of the general principle of democracy applied to our country's most urgent needs.\n",
      "French revolutionary leader Maximilien Robespierre, 1794\n",
      "With which of the following threats does the speaker in the passage above seem most concerned?\n",
      "Options:\n",
      "(A): Counterrevolutionary opponents\n",
      "(B): Foreign invaders\n",
      "(C): Communist agitators\n",
      "(D): Transnational corporations\n",
      "A:\n"
     ]
    }
   ],
   "source": [
    "print(datasets[20][\"instruction\"])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
